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I.  Introduction 

Under  certain  smoothness  and  controllability  assumptions  the  solutions 
of  broad  class  of  optimal  control  problems  is  described  by  the  Hamilton- 
Jacobi-Bellman  equation  [1] .  Under  some  further  restrictions,  the  optimal 
state-feedback  control  law  can  be  computed  analytically  once  the  solution 
of  this  equation  is  known.  In  certain  applications  (e.g.,  [2]),  the 
global  nonlinear  control  law  has  been  shown  to  offer  significant  advantages, 
in  performance  as  well  as  implementation,  over  local  linearized  control  laws. 

The  problem  of  obtaining  approximate  solutions  of  the  Hamilton-Jacobi- 
Bellman  equations,  therefore,  has  received  considerable  attention.  Quasi¬ 
linearization  [3] ,  power  series  [4] ,  and  global  analysis  techniques  [5] , 

[6],  [7],  to  mention  but  a  few  examples,  have  been  applied  to  this  problem. 

The  purpose  of  this  brief  paper  is  to  point  out  a  functional  expansion 
technique  which  is  applicable  to  the  solution  of  the  nonlinear  partial 
differential  equation  that  is  obtained  when  the  optimal  control  cam  be 
expressed  analytically  in  terms  of  the  cost-to-go.  Analytical  examples 
are  presented  which  provide  an  indication  of  the  nature  of  convergence  of 
the  method.  A  complete  numerical  analysis  of  convergence  is  not  provided 
here  (and  is  probably  best  pursued  in  the  context  of  individual  applications) ; 
but  it  is  suggested  that  the  functional  expansion  approach  is  often  better- 
suited  than  power  series  methods  for  numerical  computations. 
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II.  Nonlinear  State-Feedback  Laws  for  a  Class  of  Optimal  Control  Problems 


In  this  section,  we  give  an  explicit  statement  of  the  class  of  optimal 
control  problems  which  can  be  addressed  using  the  functional  expansion 
techniques  of  the  following  section,  and  display  the  Hamilton-Jacobi-Bellman 
(HJB)  equation  characterizing  the  optimal  control  law. 

Let  Itgft^]  denote  the  time  interval  of  the  problem  and  let  the  space 
li  ,  of  admisable  controls  be  a  convex  open  subset  of  PC([tQ,t^]»R  ),  the 


space  of  piecewise  continuous  functions  on  ltg,t^J  taking  values  in  R  . 

We  seek  an  optimal  control  u *eU  to  minimize  the  functional  J:  U"*R  defined 

*>y 


'J(u)  =  /  L(x(t)  ,u(t)  )dt 


(1) 


subject  to  the  constraints 


x  (t)  **  f  (x  (t) , u  ( t) )  (2) 

x(V  =  x0  (3) 


The  functional  L:  RnxRr  4  R  is  assumed  to  take  the  form 


L(x,u)  -  u’  u  +  Q(x)  (4) 

where  Q(x)  satisfies 


(i) 

Q(x)  =>  Q(-x), 

and  Q(x) 

>  0  for  all  xeRn,  : 

(ii) 

lim  Q(x)  = 

0;  lim 

S  >  q  >  o 

!  1  x |  |"*0 

IMI 

1-11*11  ” 

(iii) 

Q(x)  is  analytic 

in  x. 

The  function  f:  R°xRr  R°  is  assumed  to  take  the  form 
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f(x,u)  “  Ax  +  Bu  +  f(x,u)  (5) 

where 

n 

f(x,u)  -  fQ(x)  +  l  e^^u' (Flif1  (x)  +  F2i(x)u)  (6) 

i=l 

The  following  additional  assumptions  are  imposed  on  f: 

(iv)  (A,B)  is  a  completely  controllable  pair. 

(v)  ?  (x,u)  is  analytic  in  x,  for  all  ucRr. 

|f(x,u) 

llxll  +  Null) 

I  lu  I  I  "*0 

(vii)  lim  |  |f  n (x)  |  |  *  0;  lim  |  |f  (x)  |  |  «  0. 

I  lx  1 1-0  0  I  |x  |  |-0  21 

In  (6),  e^  denotes  the  ith  unit  basis  vector,  fQ:  Rr->Rn,  f^:  Rn~H*n, 

FlieRrxn,  and  F^:  Rn->Rrxr.  Under  assumptions  (iv)-(vii),  it  can  be  shown 
[8]  that  (2)  is  (uniformly)  completely  controllable.1 

Consider  the  truncated  problem  obtained  by  replacing  tQ  by  t eItQ,t^] 
and  xQ  by  x^  in  (1 )  —  (3 )  and  assume  that  there  exists  an  optimal  control 
u*  for  each  T.  Let  the  value  function 

V(xt>t):  Rnx[t(),t1l  -*■  R  (7) 

be  defined  as  the  (minimum)  value  of  the  cost-to-go,  which  is  achieved  for 
control  u*.  V  is  assumed  to  be  twice  continuously  differentiable  with  respect 
to  x  and  continuously 

1The  time-varying  case  can  also  bu  considered,  by  requiring  that  the  fore¬ 
going  properties  hold  uniformly  for  tett^.t^I. 


(vi)  lim 

I  Who  ( 


J 
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dif fercntiable  in  t. 

Then  the  value  function  satisfies  the  Karailton-Jacobi-Bellroan  equation 
for  problem  (l)-(3) 


-  min  {u'  u  +  Q(x)  +  f  (x,u)}  j  V(x,t.)  =  0  (8) 

*  u(t)  ®  1 


The  minimizing  control  is  given  in  feedback  form  by 


n 

V 

_J v 

2 

tbc . 

i-l 

i 

n 

t)  + 

l 

i-l 

(b*  (x*(t),t)  +  l  ™  (x*  (t)  *  t)F.  .  f  (x*(t))} 


(9) 


Inserting  (9)  in  (8)  and  simplifying  terms  gives  rise  to  a  partial  dif¬ 
ferential  equation  in  the  cost-to-go,  which  is  written-out  here  for  future 


references 


’ 
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*L  «  y  av_  l/i  +  i  y  —  (f  1  +f  )\ 

3t  4  L  3x  B  ^  3xi  rit li  J  \X  +  2  3x.  U2i  *21'/ 
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[■■  £ *  l  f  Vi] ♦ » ♦  r  «“♦  V 

.  i  JVlBf  !  +  I  ?  JV  2V  y  _JV  r  f  ] 

2  3x  [  2  J1  3x4  1F2i  2i’J  [  3x  ^  3xi  li'lj 

.  I  ?  2V_  r  421b  +  ?  JSLf.  F'.l. 

2  jii  3*j  L  3x  jij  5xi  1  lij 

[*  *  I  jj  £j  <FU  *  P2i>]'1  Vl 

+  ?  jl  ^  B  *  jx  £l  fiFu][I  *  2  X  fl  (Fii  +  F2i’]' 


With  the  boundary  condition  V(x,t  )  =  0 

1  (10) 

By  redefining  the  time  variable  as  x^  =  t^-t,  a  pure  initial  value  problem 
with  V(x,  x^  =  0)  =  0  is  obtained.  The  Cauchy-Kowalewski  Theorem  [9]  can 
be  applied  to  guarantee  local  existence  and  uniqueness  of  solutions  to  (10) 
about  x^  =  0;  the  foregoing  hypotheses  further  guarantee  that  such  a  solution 
can  be  uniquely  continued  to  x^  =  t^  and  thus  establish  the  smoothness  pre¬ 
requisite  for  the  functional  expansion  technique  to  be  introduced  in  the 


next  section. 
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In  the  sequel,  we  will  consider  examples  where  lira  t^  =  °°?  in  this 
case  a  solution  of  the  algebraic  equation  obtained  by  setting  the  left-hand 
side  of  (10)  to  zero  can  be  guaranteed  under  the  additional  hypothesis  that 
V  satisfies  the  conditions  of  Lyapunov's  second  theorem  [10].  Bellman  and 
Bucy  111)  explored  analytic  methods  for  this  problem  under  the  further 
assumption  that  ?(x,u)  =  0  in  (5)?  their  work  indicates  the  considerable 
complexity  of  the  nonlinear  feedback  problem. 

Equation  (10)  may  be  regarded  as  a  special  case  of  a  more  general 
class  of  problems  to  which  Bellman's  functional  expansion  technique  applies? 
we  use  it  as  motivation  to  introduce  the  more  general  notation  for  the  next 
section.  Transferring  all  terms  in  V  to  the  left-hand  side  and  identifying 
v  with  V,  and  g  with  Q,  the  only  remaining  term  on  the  right-hand  side,  we 
may  write 

Nv  =  g  ;  v(x,T^  =  0)  =  0  (li) 

where  the  nonlinear  operator  N  can  be  viewed  as  a  mapping  from  the  Banach 

space  C1'2 (Rnx[0,t1) ?  R)  into  itself,  or  more  generally  as  a  mapping  from 

2 

one  Banach  space  to  smother. 

By  virtue  of  the  preceding  assumptions^  there  is  a  related  linear- 
quadratic  control  problem  whose  solution  is  known?  the  HJ3  equation  for  it 
is  denoted 

Wovo  "  go  *  vo(x'Ti  =  0)  £  0  (12) 

2Further  definition  of  the  topology  is  omitted  because  it  would  unnecessarily 
complicate  the  presentation. 


whore 


is  the  first  non-zero  term  in  the  Taylor  series  expansion  of  Q(x) ; 
is  a  positive-definite  symmetric  matrix  by  virtue  of  assumptions  (i) 


The  main  point  of  this  brief  paper  is  that  eq.  (12),  the  natural  auxiliary 
equation  for  solving  (11)  happens  to  be  nonlinear  and  thus  a  nonlinear  ex- 


-9- 


-10- 


will  converge  as  desired. 

Bellman  {13]  introduced  a  "fundamental  technique"  for  recursively  com¬ 
puting  the  function  v^+J  in  terras  of  which  was  illustrated  by 

means  of  an  example  involving  a  linear  auxiliary  equation.  We  shall  illustrate 
the  essential  procedure  for  using  a  nonlinear  auxiliary  equation  and  show 
that  the  recursion  relation  remains  linear  in  v^+1-  Formally,  the  procedure 
is  to  substitute  the  series  expression  (15)  into  (14)  and  to  equate  terms 
in  like  powers  of  E.  Volterra  [14,  p.  24]  has  developed  an  "extension  of 
Taylor's  theorem"  to  a  class  of  functionals,  which  can  be  applied  to  show 
that  there  exist  operators  N  ,  etc.  such  that 

N  v  =  N  (v  +  e  v  +  ....)  (16) 

0  e  0  0  1 


=  Nq  vQ  +  eW1(v0)v1  +  e  N^.v^v 2  +  ... 


and  operators  A  N  ,  A  etc.,  such  that 


A  N0  ve  =  {VN)ve 


=  (N  -N)  (vQ  tEVj^  +  c  v2  +  ...) 


=  A  WQ  vQ  +  eA  W1(v0)vjL  +  e  A^  (v^v^  +  . . . 


(17) 


A  further  property  of  the  operators  N  ,  i  1  is  asserted  by  the  following 


lemma: 
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Lemma:  Under  the  conditions  described  in  [14,  p.  24],  the  operators 

N  are  affine  in  v^,  given  i-e. 


Vv-vi1'! 


UvQ)v.  +  ^  (vQ . Vl> 


This  result  is  established  in  the  appendix. 

From  (14)- (18)  we  may  then  conclude  that  the  can  be  computed 
recursively  from  the  equations 

NgVg  =  gQ  (solution  assumed  known) . 

t(VVl  '  “o’o  -  43 0  -  W  ‘  &0  -  90  -  9 

Uv0)v2  -  4N1(v0iv1  -  iyv'V  1191 


l(V\ 


"  4\-l‘V0 . Vi’Vl 


VV" 


•Vi1 


We  remark  that  for  the  case  of  a  linear  auxiliary  equation  (as  considered 
by  Bellman),  say  ■*  LQ,  then  the  operators  L  in  (19)  are  independent  of 
vQ,  and  are  in  fact  all  equal  to  LQ.  In  the  special  case  noted  previously, 
Bellman  and  Bucy  [11]  obtained  a  procedure  similar  to  (19). 

To  illustrate  the  recursion  (19) ,  we  apply  it  to  the  particular  problem 
(10)  with  the  simplifying  assumption  F  ^  =  0,  i  =  l,...,n,  and  show  that 
more  can  be  said.  The  first  equation  of  (19)  corresponds  to  (12).  By 
applying  the  Lemma,  it  can  be  shown  that  the  n-th  equation  takes  the  form 
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L(V\ 


3vk  r  i 
“57  -[7 


a/.* 


BB' 


3x  ****  3x 


.  ar  a/. 
°  *  7  17  “•  17 


]*3 


Ax 


l(V0'**"Vk  2)Vk  1  ~  ^(V0"'"Vk  1* 


(20) 


It  is  well-established  ([1],  [5],  [15])  that  (12)  has  the  solution 


VQ(x,  t)  =  x*  KQ(t)x  (21) 

with 

•  577  Ko  “  V  *  A'Ko  “  KoBB'Ko  *  2!  K(tl>  “  0  <22> 

and  thus 

3V  3yv 

lt(vo,vk]  (x,t)  =  It  <x't,  +  X’(A’  ”  Ko(t)BB’>  Vk(x't1)  H  0 

(23) 

To  find  VR(x,t)  then,  requires  the  solution  of  a  linear  variable- 
coefficient  first-order  partial  differential  equation  in  n+1  independent 
variables. 

While  a  complete  convergence  analysis  lies  beyond  the  scope  of  this 
paper,  we  merely  indicate  some  of  the  considerations  involved.  Since 
V^:Rnx [tQ,t^]  ■*  R,  some  means  of  dealing  with  the  unboundedness  of  ||x|| 
must  be  provided  if  numerical  solutions  are  to  be  considered.  One  means 
is  to  seek  a  solution  for  ||x||  <  p,  where  p  is  chosen  sufficiently  large 

to  accomodate  all  initial  conditions  of  interest;  in  this  case  it  is 
necessary  to  verify  that  the  solution  for  ||x||  £  p  at  time  t  does  not  depend 
on  the  solution  for  ||x||  >  p,  for  te[t,t^].  That  this  is  in  fact  the  case. 


can  be  seen  by 
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applying  the  method  of  characteristics  to  (20) ,  and  noting  from  (23)  that 
the  hypothesis  (iv)  of  complete  controllability  of  (A,B),  along  with  hypo¬ 
thesis  (i)-(ii),  which  imply  that  Q  is  positive  definite,  imply  further  that 
(A0-BB’KQ(t))  is  uniformly  stable,  and  thus  the  characteristics  diverge  from 
x=0  as  increases  from  0  to  t^.  Also,  it  is  to  be  expected  that 
lim  V  (x,t)  -  ~  for  each  te[t  ,t  );  hence  the  best  convergence  result 

INI  ~ 

that  might  be  expected  is  that  ||v(x,t)  ||  _<  Me  ^  for  some  values  of  M>0 

and  Ct^O  this  is  much  weaker  that  the  usual  bounds  employed  in  solving  p.d.e.'s 

Finally,  the  aforementioned  conditions  on  ||(NQ-N)v||  and  |  |g— 1 1  relate 

to  the  rate  of  growth  of  the  driving  terms  on  the  right-hand  side  of  (20) 

as  n  increases?  in  fact,  these  terms  have  been  approximated  to  third  order 

by  the  proposed  auxiliary  equation. 

A  useful  technique  that  is  exploited  extensively  in  the  examples  of 

the  following  section  is  separation  of  variables.  Certain  problems  may  be 

solved  exactly  by  this  method.  Even  in  cases  of  numerical  computation,  a 

multinomial  expansion  of  V  can  reduce  the  computations  to  solving  a 

n 

finite  system  of  ordinary  differential  equations  in  the  time-variable  alone. 

In  these  cases,  the  control  law  approximation 


(24) 
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can  also  be  expressed  as  a  power  series  in  x  (the  solution  of  (5)  with 

k 

control  u  ) j  howovor,  this  scries  is  not  in  general  the  same  as  the  power 
series  for  the  optimal  control  law  obtained  directly  from  a  scries  solution 
of  (10). 

Although  there  is  a  formal  correspondence  in  these  cases  between 
solutions  obtained  by  power  series  and  by  the  functional  expansion  technique 
it  is  thus  difficult  to  evaluate  the  relative  computational  merits  of  the 
two  procedures.  The  relative  difficulty  of  computing  the  functional  ex¬ 
pansion  operators  W^»  i  ^  1  must  be  balanced  by  the  relative  simplicity 

of  solving  the  recursion  (19) ,  for  which  standard  numerical  procedure  are 
available,  and  the  relative  ease  of  testing  for  convergence.  Further 
numerical  analysis  of  the  functional  expansion  technique  appears  warranted, 
particularly  for  problems  that  are  not  readily  amenable  to  power-series 
solutions. 


■I 
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IV.  Examples 
Example  Is 

A  scalar  control  u  on  [t  ,^1  =  [0,Tl  is  sought  to  minimize 
T 

J(u)  =  /  (u2(t)  +  X2(t)  +  ^-x4(t))dt 

0 

subject  to  the  scalar  state  equation 


(25) 


x  (t) 


u (t)  +  px  (t) 


x(t0)  “  X0 


(26) 


The  optimal  control  law  corresponding  to  (9)  is 
uMt)  -  -  j  (x*(t),t) 


(27) 


and  the  HJB  equation  corresponding  to  (10)  is: 

Sr  ■  Tdr)2-"*3^-*2  -  i1'"  iVU-T)  ■ 0  (28> 


The  system  (19)  takes  the  form 


(29) 

k  »  2,3, . . . 


. . -  - 


i 


The  solution  of  the  first  (auziliary)  equation  is  known  [17]  to  be 


V0<x,t) 


KQ (t) x 


dK0(t)  2 

sr  -  + 1 


VT) 


(30) 


so  Kq  (t) 
V1(x,t) 


tanh  (T-t) .  V^(xft)  may  be  found  by  separation  of  variables  as 

4  dKl(t>  1 

Kl(t)x  *  ~ dt —  "  ^tanMT-tjK^t)  E  ~  2  *  2Mtanh(T-t);  (T)  *•-  0 

(31) 


which  has  the  solution 


Kj(t)  =  ^  [1  -  cosh"4  (T-t)  ]  +  l~  (T-t)  +  |  sinh  2  (T-t) 


1  -4 

+  tt  sinh  4 (T-t)] cosh  (T-t) 

04 


(32) 


Similarly, 


V  <X,t)  -  Kit),6;  ^  -  «0(t,K2(t) 

*  *  at 


4Kx(t)  -  4yK1(t);  K2(T)  =  0 


(33) 


All  of  the  succeeding  equations  may  be  solved  by  separation  of  variables 
and  use  of  the  known  variation  of  constants  formula  for  the  solution  of  a 
scalar  time-varying  linear  equation.  The  optimal  control  approximation 
thus  assumes  the  form 


uk(t) 


(i+UK^tJx 


(2i+l) 


(34) 


It  should  be  noted  that  this  is  not  the  same  type  of  approximation  obtained 
from  ordinary  power  series  or  from  singular  perturbations  in  the  parameter 
)*  (if  its  value  is  small). 
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Example  2: 

A  scalar  bilinear  control  problem  on  (t^t  1  =  10»  1  is  to  minimize 

J(u)  =  /  <u2(t)  +  x2(t)  +  ^-x4(t)  +  X  (t))dt  (35) 

0 

subject  to 

x (t)  =  ax (t)  +  bu(t)  +  cu (t)x (t) ;  x(0)  =  xQ  (36) 

The  optimal  control  law  corresponding  to  (9)  is 

u*(t)  =  -  2L  (x*(t),t)  *  07) 

The  limiting  form  of  the  HJB  equation  (as  t  -*>)  is 


The  limiting  form  of  (19)  is: 
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(* 

/b 

l  2  3x 
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5TT 

‘“I 
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"  4X 

( f 

\i=0 
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<>x  3x  J 

*0 

\ 

3v 

A  n 

/  3 

3  Vi  3vi\ 

3x  3x  I 

3x 

-  ax  I 

*• 

ax 

V* 

xs  — 

4 

M  I 

\i=0 

'  3V0 
3x 

l 

> 

-  ax 

i 

1  3V* 

1  ax 

c 

4 

•  • 

/k-l 

*  1 
\  i=0 

sVi-i  3vi 

3x  3x 

k  =  5,6,7 

3V2-i  3V- 

9x  3x 


) 


+  x 


.11 

\  jMl  3x  3x  / 


.  (Y  !Vti  !!i\ 

\  Y  »«  »*  / 


(39) 


Applying  the  same  idea  as  in  the  previous  example  we  find 


Vx)  =  V2  »  bK0  ■  2K0a  "  1 


(40) 


J~ 2 

+  /a 


which  has  the  (stable)  solution  =  -■  —  +b  .  Taking 


Vx(x) 


KjX  ;  abK^  -  3Kxa 


cK 


(41) 


—1  2  v+o 

gives  *=  c  [3bKQ-3a]  K^.  Similarly,  the  coefficients  for  V^(x)  =  K^x 

can  be  identified.  The  approximate  optimal  control  is  then 


k,  x 
u  (t) 


(b+cx 
2 


)  * 

'  i=0 


(i+2)K^  x 


(i+1) 


(42) 


In  this  steady-state  case  where  power-law  solutions  for  V^(x)  can  be  assumed 
we  thus  see  that  the  coefficients  will  bo  uniquely  determined  from  the  solution 
of  linear  equations,  once  the  proper  (stabilizing)  solution  of  the  auxiliary 
equation  is  chosen. 
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V.  Conclusions 

In  this  brief  paper,  we  have  pursued  the  extension  of  a  functional 
expansion  technique  suggested  originally  by  Bellman  and  showed  that  it  has 
interesting  consequences  when  applied  to  the  solution  of  the  Ham il ton -Jacob i- 
Bellman  partial  differential  equation  for  a  certain  class  of  optimal  control 
problems.  While  this  is  perhaps  implicit  in  Bellman's  own  work,  we  consider 
it  worthwhile  to  have  clarified  the  nature  of  the  continuation  hypothesis 
involved,  to  have  identified  the  explicit  requirements  for  convergence 
(although  a  formal  proof  has  not  been  provided) ,  and  to  have  more  clearly 
delineated  the  class  of  problems  where  the  technique  is  potentially  most 
useful.  Furthermore,  we  have  distinguished  this  technique  from  power 
series  methods  which  have  been  more  commonly  applied,  but  which  can  yield 
inferior  solutions  to  highly  nonlinear  or  time-varying  problems. 


I  . 

.  . 


-20- 


Appendix  ;  Demonstration  of  I.emma 

Given  a  nonlinear  and  sufficiently  regular  operator  operating  on  a 
convergent  power  series 

OO 

v£  =  J  v^1  (43) 

i=0 

it  is  to  be  demonstrated  that  the  functional  expansion 


.L  V\ 


i=0 


•v.  ,)v.  c 
r-1  l 


(44) 


has  affine  terms 


Vv— Vi,vi 


l<v0lvi  *  Vvo . vi-l> 


This  can  be  seen  by  direct  recourse  to  Vol terra's  definitions  of  the 
operators  p.24): 


Ni(vQ, . . .v 


)v  =  (v ) 

-1  1  i!  Ldc" J  ° 


1  ,  2  ,  •  •  • 


(45) 


e=o 


For  instance  (i=l) 


N  (v  )v  -  I  ( 

l'V  1  J  jn 


&H 


t0  R 


o(vo) 

<Sv 


lx,t]  v^(x,t)dxdt 


(46) 


where  6NQ/6vQ  is  the  functional  (Frechet)  derivative  of  NQ  with  respect 
to  v  at  vQ  and  evaluated  at  [x,t]e  R°xlt  ,t  ].  This  is  seen  to  be  a  linear 
operation  on  v  . 


For  1=2,  we  find 
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